Systems and methods to provide audible output based on section of content being presented

ABSTRACT

A device provides audible output pertaining to audio video (AV) content such as a video game based on a section of the AV content that is being presented.

FIELD

The application relates generally to technically inventive, non-routinesolutions that are necessarily rooted in computer technology and thatproduce concrete technical improvements.

BACKGROUND

Video games are a fun and engaging source of entertainment for manypeople. However, playing video games can sometimes be frustrating to auser if the user cannot complete a certain objective within the videogame or is having trouble even identifying the objective to complete.There are currently no adequate solutions to the foregoingcomputer-related, technological problem.

SUMMARY

Accordingly, in one aspect a device includes at least one processor andat least one computer storage with instructions executable by the atleast one processor. The instructions are executable to identify alocation within audio video (AV) content that is currently beingpresented and to provide, based on the identification of the locationwithin the AV content, audible assistance pertaining to the AV content.

In some implementations, the AV content may include a video game, andthe location may be a chapter or sub-chapter of the video game. Thelocation may be identified based at least in part on images from acamera in communication with the at least one processor, with the imagesfrom the camera indicating a portion of the AV content.

Also, in some embodiments the audible assistance may be first audibleassistance, the location may be a first location, and the instructionsmay be executable by the at least one processor to provide the firstaudible assistance based on the identification of the first locationmeeting a threshold confidence level and to decline to provide secondaudible assistance based on identification of a second location withinthe AV content not meeting the threshold confidence level.

The content of the audible assistance may vary based on a level ofconfidence in the identification of the location. For example, thecontent of the audible assistance may be tailored to pertain to thelocation based on the level of confidence in the identification of thelocation being above a threshold confidence level and may not betailored to pertain to the location based on the level of confidence inthe identification of the location being below the threshold confidencelevel.

In some implementations the content of the audible assistance may bedetermined based on execution of natural language processing on speechreceived from a user to identify information relevant to what is spokenby the user. Additionally or alternatively, the content of the audibleassistance may be tailored to a user's emotion(s) while observing the AVcontent.

In examples where the AV content is a video game, the content of theaudible assistance may even be tailored to one or more of a player'sgameplay ability and a difficulty level at which the video game is set.Additionally or alternatively, the content of the audible assistance mayvary based on an objective of a player of the video game, where theobjective may pertain to one or more of a game score, a player rating,and a level of engagement with various aspects of the video game.

In some embodiments, the device may be a first device established by arobot_(;) with the robot being different from a second device at whichthe AV content is presented. Also in some embodiments, the device may beembodied in a video game console or another device that communicateswith the video game console to identify the location.

In another aspect, a computer-implemented method includes identifying asection of content that is currently being presented via at least onedevice and providing, based on the section of the content that iscurrently being presented, audible assistance pertaining to the section.

In still another aspect, an apparatus includes at least one computermemory' that includes instructions executable by at least one processorto identify a section of a video game that is currently being presentedvia a device and to provide, based on the section of the video game thatis currently being presented, audible assistance pertaining to the videogame.

The details of the present application, both as to s structure andoperation, can best be understood in reference to the accompanyingdrawings, in which like reference numerals refer to like parts, and inwhich:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example system consistent with presentprinciples;

FIGS. 2-7 are example illustrations consistent with present principles;

FIGS. 8 and 9 are flow charts of example logic consistent with presentprinciples; and

FIG. 10 shows an example graphical user interface consistent withpresent principles.

DETAILED DESCRIPTION

This disclosure relates generally to computer ecosystems includingaspects of consumer electronics (CE) device networks such as but notlimited to distributed computer game networks, augmented reality (AR)networks, virtual reality (VR) networks, video broadcasting, contentdelivery networks, virtual machines, and machine learning applications.

A system herein may include server and client components, connected overa network such that data may be exchanged between the client and servercomponents. The client components may include one or more computingdevices including AR headsets, VR headsets, game consoles such as SonyPlayStation® and related motherboards, game controllers, portabletelevisions (e.g. smart TVs, Internet-enabled TVs), portable computerssuch as laptops and tablet computers, and other mobile devices includingsmart phones and additional examples discussed below. These clientdevices may operate with a variety of operating environments. Forexample, some of the client computers may employ, as examples, Orbis orLinux operating systems, operating systems from Microsoft, or a Unixoperating system, or operating systems produced by Apple, Inc. orGoogle. These operating environments may be used to execute one or moreprograms/applications, such as a browser made by Microsoft or Google orMozilla or other browser program that can access websites hosted by theInternet servers discussed below. Also, an operating environmentaccording to present principles may be used to execute one or morecomputer game programs/applications and other programs/applications thatundertake present principles.

Servers and/or gateways may include one or more processors executinginstructions that configure the servers to receive and transmit dataover a network such as the Internet. Additionally or alternatively, aclient and server can be connected over a local intranet or a virtualprivate network. A server or controller may be instantiated by a gameconsole and/or one or more motherboards thereof such as a SonyPlayStation®, a personal computer, etc.

Information may be exchanged over a network between the clients andservers. To this end and for security, servers and/or clients caninclude firewalls, load balancers, temporary storages, and proxies, andother network infrastructure for reliability and security. One or moreservers may form an apparatus that implement methods of providing asecure community such as an online social website or video game websiteto network users to communicate crowdsourced in accordance with presentprinciples.

As used herein, instructions refer to computer-implemented steps forprocessing information in the system. Instructions can be implemented insoftware, firmware or hardware and include any type of programmed stepundertaken by components of the system.

A processor may be any conventional general-purpose single- ormulti-chip processor that can execute logic by means of various linessuch as address lines, data lines, and control lines and registers andshift registers.

Software modules described by way of the flow charts and user interfacesherein can include various sub-routines, procedures, etc. Withoutlimiting the disclosure, logic stated to be executed by a particularmodule can be redistributed to other software modules and/or combinedtogether in a single module and/or made available in a shareablelibrary.

As indicated above, present principles described herein can beimplemented as hardware, software, firmware, or combinations thereof;hence, illustrative components, blocks, modules, circuits, and steps areset forth in terms of their functionality.

Further to what has been alluded to above, logical blocks, modules, andcircuits described below can be implemented or performed with ageneral-purpose processor, a digital signal processor (DSP), a fieldprogrammable gate array (FPGA) or other programmable logic device suchas an application specific integrated circuit (ASIC), discrete gate ortransistor logic, discrete hardware components, or any combinationthereof designed to perform the functions described herein. A processorcan be implemented by a controller or state machine or a combination ofcomputing devices.

The functions and methods described below may be implemented in hardwarecircuitry or software circuitry. When implemented in software, thefunctions and methods can be written in an appropriate language such asbut not limited to Java, C# or C++, and can be stored on or transmittedthrough a computer-readable storage medium such as a random accessmemory (RAM), read-only memory (ROM), electrically erasable programmableread-only memory (EEPROM), compact disk read-only memory (CD-ROM) orother optical disk storage such as digital versatile disc (DVD),magnetic disk storage or other magnetic storage devices includingremovable thumb drives, etc. A connection may establish acomputer-readable medium. Such connections can include, as examples,hard-wired cables including fiber optics and coaxial wires and digitalsubscriber line (DSL) and twisted pair wires. Such connections mayinclude wireless communication connections including infrared and radio.

Components included in one embodiment can be used in other embodimentsin any appropriate combination. For example, any of the variouscomponents described herein and/or depicted in the Figures may becombined, interchanged or excluded from other embodiments.

“A system having at least one of A, B, and C” (likewise “a system havingat least one of A, B, or C” and “a system having at least one of A, B,C”) includes systems that have A alone, B alone, C alone, A and Btogether, A and C together, B and C together, and/or A, B, and Ctogether, etc.

Now specifically referring to FIG. 1, an example system 10 is shown,which may include one or more of the example devices mentioned above anddescribed further below in accordance with present principles. The firstof the example devices included in the system 10 is a consumerelectronics (CE) device such as an audio video device (AVD) 12 such asbut not limited to an Internet-enabled TV with a TV tuner (equivalently,set top box controlling a TV). However, the AVD 12 alternatively may bean appliance or household item, e.g. computerized Internet enabledrefrigerator, washer, or dryer. The AVD 12 alternatively may also be acomputerized Internet enabled (“smart”) telephone, a tablet computer, anotebook computer, an augmented reality (AR) headset, a virtual reality(VR) headset, Internet-enabled or “smart” glasses, another type ofwearable computerized device such as a computerized Internet-enabledwatch, a computerized Internet-enabled bracelet, a computerizedInternet-enabled music player, computerized Internet-enabled headphones, a computerized Internet-enabled implantable device such as animplantable skin device, other computerized Internet-enabled devices,etc. Regardless, it is to be understood that the AVD 12 is configured toundertake present principles (e.g., communicate with other consumerelectronics (CE) devices to undertake present principles, execute thelogic described herein, and perform any other functions and/oroperations described herein)

Accordingly, to undertake such principles the AVD 12 can be establishedby some or all of the components shown in FIG. 1. For example, the AVD12 can include one or more displays 14 that may be implemented by a highdefinition or ultra-high definition “4K” or higher flat screen and thatmay be touch-enabled for receiving user input signals via touches on thedisplay. The AVD 12 may include one or more speakers 16 for outputtingaudio in accordance with present principles, and at least one additionalinput device 18 such as an audio receiver/microphone for enteringaudible commands to the AVD 12 to control the AVD 12. The example AVD 12may also include one or more network interfaces 20 for communicationover at least one network 22 such as the Internet, an WAN, an LAN, etc.under control of one or more processors. Thus, the interface 20 may be,without limitation, a Wi-Fi transceiver, which is an example of awireless computer network interface, such as but not limited to a meshnetwork transceiver. Furthermore, note the network interface 20 may be,e.g., a wired or wireless modem or router, or other appropriateinterface such as, for example, a wireless telephony transceiver, orWi-Fi transceiver as mentioned above, etc.

It is to be understood that the one or more processors control the AVD12 to undertake present principles, including the other elements of theAVD 12 described herein such as con rolling the display 14 to presentimages thereon and receiving input therefrom. The one or more processorsmay include a central processing unit (CPU) 24 as well as a graphicsprocessing unit (GPU) 25 on a graphics card 25A.

In addition to the foregoing, the AVD 12 may also include one or moreinput ports 26 such as, e.g., a high definition multimedia interface(HDMI) port or a USB port to physically connect (e.g., using a wiredconnection) to another consumer electronics (CE) device and/or aheadphone port to connect headphones to the AVD 12 for presentation ofaudio from the AVD 12 to a user through the headphones. For example, theinput port 26 may be connected via wire or wirelessly to a cable orsatellite source 26 a of audio video content. Thus, the source 26 a maybe, e.g., a separate or integrated set top box, or a satellite receiver.Or, the source 26 a may be a game console or disk player containingcontent that might be regarded by a user as a favorite for channelassignation purposes. The source 26 a when implemented as a game consolemay include some or all of the components described below in relation tothe CE device 44 and may implement some or all of the logic describedherein.

The AVD 12 may further include one or more computer memories 28 such asdisk-based or solid-state storage that are not transitory signals, insome cases embodied in the chassis of the AVD as standalone devices oras a personal video recording device (PVR) or video disk player eitherinternal or external to the chassis of the AVD for playing back AVprograms or as removable memory media. Also in some embodiments, the AVD12 can include a position or location receiver such as but not limitedto a cellphone receiver, GPS receiver and/or altimeter 30 that isconfigured to, e.g., receive geographic position information from atleast one satellite or cellphone tower and provide the information tothe processor 24 and/or determine an altitude at which the AVD 12 isdisposed in conjunction with the processor 24. However, it is to beunderstood that that another suitable position receiver other than acellphone receiver, GPS receiver and/or altimeter may be used inaccordance with present principles to, for example, determine thelocation of the AVD 12 in all three dimensions.

Continuing the description of the AVD 12, in some embodiments the AVD 12may include one or more cameras 32 that may be, e.g., a thermal imagingcamera, a digital camera such as a webcam, an infrared (IR) camera,and/or a camera integrated into the AVD 12 and controllable by theprocessor 24 to generate pictures/images and/or video in accordance withpresent principles. Also included on the AVD 12 may be a Bluetoothtransceiver 34 and other Near Field Communication (NFC) element 36 forcommunication with other devices using Bluetooth and/or NFC technology,respectively. An example NFC element can be a radio frequencyidentification (RFID) element.

Further still, the AVD 12 may include one or more auxiliary sensors 37(e.g., a motion sensor such as an accelerometer, gyroscope, cyclometer,or a magnetic sensor, an infrared (IR) sensor, an optical sensor, aspeed and/or cadence sensor, a gesture sensor (e.g., for sensing gesturecommand), etc.) providing input to the processor 24. The AVD 12 mayinclude an over-the-air TV broadcast port 38 for receiving OTA TVbroadcasts providing input to the processor 24. In addition to theforegoing, it is noted that the AVD 12 may also include an infrared (IR)transmitter and/or IR receiver and/or IR transceiver 42 such as an IRdata association (IRDA) device. A battery (not shown) may be providedfor powering the AVD 12.

Still referring to FIG. 1, in addition to the AVD 12, the system 10 mayinclude one or more other consumer electronics (CE) device types. In oneexample, a first CE device 44 may be used to send computer game audioand video to the AVD 12 via commands sent directly to the AVD 12 and/orthrough the below-described server while a second CE device 46 mayinclude similar components as the first CE device 44. In the exampleshown, the second CE device 46 may be configured as an AR or VR headsetworn by a user 47 as shown. In the example shown, only two CE devices44, 46 are shown, it being understood that fewer or greater devices mayalso be used in accordance with present principles.

In the example shown, to illustrate present principles all three devices12, 44, 46 are assumed to be members of a network such as a secured orencrypted network, an entertainment network or Wi-Fi in, e.g., a home,or at least to be present in proximity to each other in a certainlocation and able to communicate with each other and with a server asdescribed herein. However, present principles are not limited to aparticular location or network unless explicitly claimed otherwise.

The example non-limiting first CE device 44 may be established by anyone of the above-mentioned devices, for example, a smart phone, adigital assistant, a portable wireless laptop computer or notebookcomputer or game controller (also referred to as “console”), andaccordingly may have one or more of the components described below. Thesecond CE device 46 without limitation may be established by an ARheadset, a VR headset, “smart” Internet-enabled glasses, or even a videodisk player such as a Blu-ray player, a game console, and the like.Still further, in some embodiments the first CE device 44 may be aremote control (RC) for, e.g., issuing AV play and pause commands to theAVD 12, or it may be a more sophisticated device such as a tabletcomputer, a game controller communicating via wired or wireless linkwith a game console implemented by another one of the devices shown inFIG. 1 and controlling video game presentation on the AVD 12, a personalcomputer, a wireless telephone, etc.

Accordingly, the first CE device 44 may include one or more displays 50that may be touch-enabled for receiving user input signals via toucheson the display 50. Additionally or alternatively, the display(s) 50 maybe an at least partially transparent display such as an AR headsetdisplay or a “smart” glasses display or “heads up” display, as well as aVR headset display, or other display configured for presenting AR and/orVR images.

The first CE device 44 may also include one or more speakers 52 foroutputting audio in accordance with present principles, and at least oneadditional input device 54 such as, for example, an audioreceiver/microphone for entering audible commands to the first CE device44 to control the device 44. The example first CE device 44 may furtherinclude one or more network interfaces 56 for communication over thenetwork 22 under control of one or more CE device processors 58. Thus,the interface 56 may be, without limitation, a Wi-Fi transceiver, whichis an example of a wireless computer network interface, including meshnetwork interfaces. It is to he understood that the processor 58controls the first CE device 44 to undertake present principles,including the other elements of the first CE device 44 described hereinsuch as, e.g., controlling the display 50 to present images thereon andreceiving input therefrom. Furthermore, note that the network interface56 may be, for example, a wired or wireless modem or router, or otherappropriate interface such as a wireless telephony transceiver, or Wi-Fitransceiver as mentioned above, etc.

Still further, note that in addition to the processor(s) 58, the firstCE device 44 may also include a graphics processing unit (GPU) 55 on agraphics card 55A. The graphics processing unit 55 may he configuredfor, among other things, presenting AR and/or VR images on the display50.

In addition to the foregoing, the first CE device 44 may also includeone or more input ports 60 such as, e.g., a HDMI port or a USB port tophysically connect (e.g., using a wired connection) to another CE deviceand/or a headphone port to connect headphones to the first CE device 44for presentation of audio from the first CE device 44 to a user throughthe headphones. The first CE device 44 may further include one or moretangible computer readable storage medium 62 such as disk-based orsolid-state storage. Also in some embodiments, the first CE device 44can include a position or location receiver such as but not limited to acellphone and/or GPS receiver and/or altimeter 64 that is configured to,e.g., receive geographic position information from at least onesatellite and/or cell tower, using triangulation, and provide theinformation to the CE device processor 58 and/or determine an altitudeat which the first CE device 44 is disposed in conjunction with the CEdevice processor 58. However, it is to be understood that that anothersuitable position receiver other than a cellphone and/or GPS receiverand/or altimeter may be used in accordance with present principles to,e.g., determine the location of the first CE device 44 in all threedimensions.

Continuing the description of the first CE device 44, in someembodiments the first CE device 44 may include one or more cameras 66that may be, e.g., a thermal imaging camera, an IR camera, a digitalcamera such as a webcam, and/or another type of camera integrated intothe first CE device 44 and controllable by the CE device processor 58 togenerate pictures/images and/or video in accordance with presentprinciples. Also included on the first CE device 44 may be a Bluetoothtransceiver 68 and other Near Field Communication (NFC) element 70 forcommunication with other devices using Bluetooth and/or NFC technology,respectively. An example NFC element can be a radio frequencyidentification (RFID) element.

Further still, the first CE device 44 may include one or more auxiliarysensors 72 (e.g., a motion sensor such as an accelerometer, gyroscope,cyclometer, or a magnetic sensor, an infrared (IR) sensor, an opticalsensor, a speed and/or cadence sensor, a gesture sensor (e.g., forsensing gesture command), etc.) providing input to the CE deviceprocessor 58. The first CE device 44 may include still other sensorssuch as, for example, one or more climate sensors 74 (e.g., barometers,humidity sensors, wind sensors, light sensors, temperature sensors,etc.) and/or one or more biometric sensors 76 providing input to the CEdevice processor 58. In addition to the foregoing, it is noted that insome embodiments the first CE device 44 may also include an infrared(IR) transmitter and/or IR receiver and/or IR transceiver 78 such as anIR data association (IRDA) device. A battery (not shown) may be providedfor powering the first CE device 44. The CE device 44 may communicatewith the AVD 12 through any of the above-described communication modesand related components.

The second CE device 46 may include some or all of the components shownfor the CE device 44. Either one or both CE devices may be powered byone or more batteries.

Now in reference to the afore-mentioned at least one server 80, itincludes at least one server processor 82, at least one tangiblecomputer readable storage medium 84 such as disk-based or solid-statestorage. In an implementation, the medium 84 includes one or more solidstate storage drives (SSDs). The server also includes at least onenetwork interface 86 that allows for communication with the otherdevices of FIG. 1 over the network 22, and indeed may facilitatecommunication between servers and client devices in accordance withpresent principles. Note that the network interface 86 may be, e.g., awired or wireless modern or router, Wi-Fi transceiver, or otherappropriate interface such as a wireless telephony transceiver. Thenetwork interface 86 may be a remote direct memory access (RDMA)interface that directly connects the medium 84 to a network such as aso-called “fabric” without passing through the server processor 82. Thenetwork may include an Ethernet network and/or fiber channel networkand/or InfiniBand network. Typically, the server 80 includes multipleprocessors in multiple computers referred to as “blades” that may bearranged in a physical server “stack”.

Accordingly, in some embodiments the server 80 may be an Internet serveror an entire “server farm”, and may include and perform “cloud”functions such that the devices of the system 10 may access a “cloud”environment via the server 80 in example embodiments for, e.g., digitalor game assistant applications as disclosed herein. Additionally oralternatively, the server 80 may be implemented by one or more gameconsoles or other computers in the same room as the other devices shownin FIG. 1 or nearby.

Now in reference to FIG. 2, it shows an example illustration 200 inaccordance with present principles. It shows a player 202 of a videogame 204 that is being presented on an audio video device (AVD) 206 suchas a television under control of a video game console 208. The player202 is shown sitting on a couch 210, with the player understood to beusing a video game controller to play the video game 204, though notshown from the angle behind the player 202 that is depicted in theillustration.

As shown by speech bubble 212, the player 202 speaks “I wonder who thatis” to himself or herself in relation to a zombie 214 that is presentedas part of the video game 204. A stand-alone autonomous robot 216 isalso shown, with the robot 216 understood to be executing a digitalassistant application to receive the speech through a microphonedisposed on the robot 216 and to execute natural language processing onthe speech to identify information useful in generating audibleassistance to the player 202.

Also in order to provide relevant audible assistance, the robot 216actuates its camera 218 to gather one or more images of the video game204 as presented on the AVD 206 to identify the zombie 214 using objectrecognition and/or a comparison to pre-stored images of characters ofthe video game 204 to which the robot 216 has access.

Then, based on the zombie 214 being identified (as well as based onother potential identifications such as the zombie 214 being the onlyvideo game object in the foreground of the segment of video gamecurrently being played or being the only video game character fightingthe user's own video game character), the robot 216 may identify thesubject being referenced by the user in the user's natural languagespeech. In response, the robot 216 may provide audible assistance thatis tailored to the user's natural language speech, which in this case isto identify the zombie 214 as the zombie leader and telling the player202 to shoot the zombie 214, as represented by speech bubble 220.Additionally, though not shown in FIG. 2 it is to be understood that therobot 216 may provide visual assistance as well, such as activating alaser pointer on the robot 216 and directing laser light toward thezombie 214 as presented on display of the AVD 206 to provide a visualaid of what the robot 216 is referencing via its audible assistance.

Note that in some instances, the robot 216 may not be able toparticularly identify the zombie 214 as the zombie leader to a thresholdlevel of confidence if, for example, the zombie 214 as identified usingobject recognition might potentially match more than one reference videogame character in data to which the robot 216 has access, or if asunlight glare on the display of the AVD 204 results in the robot 216only being able to “see” part of the zombie 214 using its camera. Analterative example to FIG. 2 is shown in the illustration 300 of FIG. 3and demonstrates this.

As shown in FIG. 3, owing to the threshold level of confidence not beingmet, the robot 218 may give more-general audible assistance to theplayer 202 that does not identify the zombie 214 as the zombie leader inparticular. However, if the robot 216 were still able to identify thezombie 214 as a zombie of some kind, and based on the robot 216 havingaccess to data indicating that all zombies in the video game 204 shouldbe targeted and killed, the robot 216 may still provide some audibleassistance that is tailored to the natural language speech of the player202. In this case, the audible assistance from the robot 216 stillreferences the zombie 214 as identified as the subject of the naturallanguage speech but in this case indicates that the zombie 214 “lookslike he probably should die” even though the robot 216 does not indicatethe role of the zombie 214 as the zombie leader.

FIG. 4 shows yet another example. In the illustration 400 that is shown,the player 202 again speaks to himself or herself, wondering “who thatis” as again represented by speech bubble 212. But differentiating FIG.4 from FIG. 2, no robot is present here and an audible/digital assistantapplication for providing audible assistance is instead finning on theAVD 206, console 208, or another device in communication with either ofthose devices such as the user's smart phone. Based on the speech beingdetected using a microphone also located on one of those devices orelsewhere within the player's environment, and based on images from acamera positioned on one of those devices or elsewhere within theenvironment to gather images of the display of the AVD 206, the sametype of process(es) described above may be executed by the AVD 206,console 208, and/or other device rather than by the robot 216.Accordingly, speech bubble 402 indicates that speakers on the AVD 206may be used to provide the same audible assistance the robot 216 hadprovided in the example shown in FIG. 2.

FIG. 5 shows another example embodiment like FIG. 4 in which a “virtual”audible/digital assistant may be used rather than a separate,stand-alone autonomous robot. As shown in the illustration 500, adifferent scene, chapter, or sub-chapter from the video game 204 isbeing presented via the AVD 206. In this example, the differentscene/chapter depicts two blue-haired zombies 502, which prompts theplayer 202 to exclaim, “Two blue-haired zombies? Cool!” By processingthis natural language, the audible assistant application being executedby one of the devices within the user's environment may identify “bluehaired zombies” as the subject of the speech and look up data to whichit has access that indicates which video game chapter/sub-chapter theblue-haired zombies are encountered in the video game. The data itselfmay be accessed as stored remotely on a server by, e.g., a manufacturerof the console 208 or producer of the game 204. However, crowd-sourceddata may also be used in addition to or in lieu of the foregoing.

In any case, in the example shown in FIG. 5 the blue-haired zombies areidentified as encountered in the penultimate chapter of the video game.Then, based on the audible assistant application matching blue-hairedzombies to the penultimate chapter, the application may provide audibleassistance to the player 202 that is tailored to encourage the player202. In this case, the audible assistance indicates that the player 202is one chapter away from fighting the zombie leader at the end of thevideo game and also encourages the player 202 to kill the blue-hairedzombies to get to the final chapter, as represented by speech bubble504.

FIG. 6 shows yet another example illustration 600 in accordance withpresent principles. In illustration 600 the video game 204 is shown aspresenting a “You died!” message 602 based on the character being playedby the player 202 dying in the video game 204. In response to themessage 602, the player exclaims “Darn it!” as represented by speechbubble 604. In turn, the assistant application executing at the console,smart phone, etc. picks up on this natural language and processes it toidentify an emotion of the user using emotion recognition software or arelational database associating various exclamations with certainrespective emotions. Once an emotion has been identified (in thisexample, frustration), the assistant application may tailor audibleoutput to counteract the emotion. In this case, audible output isgenerated to assuage the player 202 by indicating, “Calm down, you haveone more life and you've almost beat the game!”, as represented byspeech bubble 606.

FIG. 7 shows yet another example illustration 700 in accordance withpresent principles. As shown, the player 202 is still playing the videogame 204. But now a rock or boulder 702 is being presented as part ofthe video game 204 at a particular location within the video game. Thelocation may be, e.g., at a particular sub-chapter of the game such as achapter beginning segment, chapter middle segment, or chapter endsegment. The audible assistant application may identify the rock/boulder702 by analyzing a frame buffer of the video game 204 provided by theconsole 208 to identify the object as a rock/boulder known to bedisposed within a certain section of the video game. Additionally oralternatively, camera input and object recognition may be used toidentify the rock/boulder 702 similar to as described above in referenceto FIG. 2 for the zombie 214. Also, an application programming interface(API) provided with the video game 204 by the video game producer andexecuting at the console 208 as part of the video game 204 may providedata indicating the rock/boulder 702, a location within the video gameat which the rock/boulder 702 appears, and a video game featureassociated with the rock/boulder 702. For example, the featureassociated with the rock/boulder 702 may be a zombie-killing weapon thatis hidden under the rock/boulder 702.

Thus, after having identified the rock/bounder 702 and the associatedfeature via the API or via other data to which the audible assistantapplication has access, the audible assistant application may provideaudible assistance to the player 202 that encourages the player 202 tolook under the rock using the video game character being controlled bythe player 202 in order to get the “full experience” of the video game.This is indicated by speech bubble 704. The application may havetailored this audible assistance based on a setting for the applicationthat has been turned on for providing the player 202 with the “fullexperience”, such as providing assistance that helps the player 202engage with various aspects of the video game that may not be criticalor necessary to finishing or beating the video game but that stillenhance gameplay and entertainment. Additionally or alternatively,machine learning may be used be used to determine that the player 202wants the “full experience” and/or the assistant application maydetermine as much based on receiving a voice command from the player 202requesting that assistance be provided to help the player 202 have the“full experience”.

Continuing the detailed description in reference to FIG. 8, it shows aflow chart of example overall logic consistent with present principlesthat may be executed by a device as part of an audible/digital assistantapplication, whether the application is executing in a stand-alone robotsuch as the robot 216 or is being executed by another device within anetwork of devices being used to undertake present principles. Beginningat block 800, the device may receive data from an API for audio video(AV) content such as a video game. The API may be running on a videogame console and developed by the provider of the video game. The APImay indicate a location/segment within the AV content that is currentlybeing presented as well as, e.g., how successful a user has been atplaying the video game, which in turn can be used to tailor audibleoutput in accordance with present principles. For example, a producer ofthe AV content and accompanying API may program the API with knownaverage times to complete a given chapter or task within a video game asdetermined based on sample gameplay data/training by video game testingpeople. The API may then, while the user plays the video game, maycompare the user's time for completing a given chapter/task to theaverage time to determine whether the user is doing better than average,average, or worse than average.

Then logic may then move to block 802 where the device may receive inputfrom one or more cameras on or otherwise in communication with thedevice. Input from one such camera may show partial or full images offrames of the AV content as presented on an AVD while input from anothercamera oriented to image the user's face may also be received to performemotion recognition using the input to thus identify an emotion of theuser in accordance with present principles.

The logic may then proceed to block 804 where the device may receiveframe buffer data from a source of the AV content, such as a video gameconsole. In some embodiments the frame buffer data may be used toidentify the location/segment within the AV content that is currentlybeing presented. Thus, the frame buffer data may indicate image framesof the video portion of the AV content so that the location/segment maybe identified, and/or the frame buffer data may itself indicate locationinformation for the location/segment currently being presented.

From block 804 the logic may then proceed to block 806 where the devicemay receive audible input from the user via a microphone on or otherwisein communication with the device. As described herein, this input may beused for a variety of purposes, such as identifying a location/segmentwithin the AV content, identifying natural language for which assistancerelated to the AV content may be provided, identifying an emotion of theuser, etc.

Thereafter the logic may proceed to block 808. At block 808 the devicemay execute natural language processing on the user's speech as receivedvia the microphone input at block 806 in order to identify, e.g., thecurrent location/segment within the AV content s well as parameters fromthe speech that may be used for tailoring audible assistance for theuser as set forth herein.

From block 808 the logic may proceed to block 810. At block 810 thedevice may identify the location/section within the AV content to afirst level of confidence based on one or more of the inputs/datareceived at blocks 800-806. For example, the device may identify thelocation/section based on objects in the foreground or background of avideo game scene as indicated in images showing the video portion of theAV content. The first level of confidence itself may be determined usingan estimation model such as a fuzzy logic algorithm, neural networkmodel, and/or a Bayesian statistics model. Artificial intelligencesoftware may also be used.

The logic may then proceed to decision diamond 812 where the device maydetermine whether the first level of confidence with which the devicehas identified the location/section is at or above a threshold level ofconfidence that may be predefined by a provider of the audible assistantapplication. Responsive to a negative determination at diamond 812, thelogic may proceed to block 814 where the device may providenon-location-specific audible assistance such as general words ofencouragement (e.g., “You're doing good, keep playing the game!”) ordecline to provide any audible assistance at all.

However, responsive to an affirmative determination at diamond 812 thelogic may instead proceed to block 816. At block 816 the device maytailor audible assistance based on the identified location/sectionwithin the AV content. In some embodiments, the audible assistance mayalso be tailored based on the first level of confidence with e.g., morespecific or descriptive assistance being provided the higher the levelof confidence in the identification of the location. The audibleassistance may also be tailored based on other factors as set forthherein, such as a user's natural language speech, a user's gameplayability, a particular difficulty level at which the video game has beenset (e.g., beginner, intermediate, expert), a user's identifiedemotions, one or more identified objectives of the user, the user'sability to complete a given video game task within an average time, etc.

Furthermore, a confidence level in the tailored output may also beevaluated by the device in some embodiments. For instance, if the devicegenerates audible assistance having certain information, but then thedevice determines that this audible assistance cannot be determined tobe appropriate or relevant to at least a threshold confidence level,then the device may generate more-general assistance and again evaluatewhether that more-general assistance is appropriate to at least thethreshold confidence level. Then once this confidence level has beenmet, the logic may proceed to block 818. At block 818 the device mayprovide the audible assistance pertaining to the AV content, such asover a speaker on a stand-alone robot, a speaker on a television beingused to present the AV content, the speaker of the user's smart phone,etc.

Now in reference to FIG. 9, another flow chart is shown of example logicthat may be executed by a device in accordance with present principles.The logic of FIG. 9 may be executed by a device executing an audibleassistant application in accordance with present principles to identifya location/segment within AV content based on a user's speech as well asinput from a camera that is imaging a display on which video of the AVcontent is presented. For the example to be described in reference toFIG. 9, it is to be assumed that the centerline axis of the field ofview of the camera is not orthogonal to the display itself but is offsetat an oblique angle.

It is to also be assumed that sunlight is reflecting off of the displayat an angle that interferes with the camera such that images generatedby the camera show only partial visual images of frames of the videocontent currently being presented on the display. For example, the glarefrom the sunlight may result in video frames presented on a left half ofthe display being represented in the input from the camera while a righthalf of the same video frames as presented on the right half of thedisplay are not shown in the camera input owing to the glare obstructingtheir view.

In any case, the logic of FIG. 9 begins at block 900 where the devicemay receive first images from the camera that indicate partial secondimages of the AV content itself as presented on the display. The logicmay then proceed to block 902 where the device may select or extract thepartial second images from the first images using image-processingsoftware.

After block 902 the logic may proceed to block 904. At block 904 thedevice may process the partial second images using the image processingsoftware to alter the partial second images as extracted in order toshow the video frames indicated in the second images from theperspective of a viewing angle/axis that is orthogonal to the display onwhich the AV content is presented. In other words, the device may “warp”the images to change the viewing angle so that a more accuratecomparison to one or more training or reference images may be made inorder to identify a location of the AV content.

The logic of FIG. 9 may then proceed to block 906, where the device mayreceive audio data via a microphone on or otherwise in communicationwith the device. The audio data may indicate natural language speech ofthe user or audio of the AV content itself. Then at block 908 the devicemay select a segment of the received audio data to identify words,phrases, or sounds from the user or AV content.

From block 908 the logic may proceed to block 910 where the device mayaccess data from a manufacturer of the device or manufacturer of aparticular video game console being used to present the AV content.Additionally or alternatively, at block 910 the device may accesscrowd-sourced data stored by the manufacturer once crowdsourced orotherwise stored at a storage location accessible to the device. Thedata itself may indicate scene or chapter changes within the AV contentand associated images associated with before or after the change, aswell as key words or sounds in the AV content from before or after thechange, as reported by the manufacturer or other users in the case ofcrowd-sourced data. The associated images, key words, or sounds may thenbe compared to the partial images and/or selected audio segment at block912 to then identify a location within the AV content at block 914 basedthe partial images/selected audio segment being matched to a givenlocation indicated in the data.

Though not shown in FIG. 9, in some embodiments the device may undertakeadditional steps after block 914. For instance, once a game location isidentified from a selected segment of audio of the AV content or fromthe partial second images, the device undertaking the logic of FIG. 9may report association of that audio/partial image frames with aparticular location within the AV content to other devices across theworld that are also executing their own instances of the audibleassistant application to help those applications make similar locationdeterminations. The device undertaking the logic of FIG. 9 may evenabstract out where scenes of the AV content are changing and report thatto other audible assistant applications too.

Reference is now made to FIG. 10, which shows an example graphical userinterface (GUI) 1000 that may be presented on a display accessible to adevice undertaking present principles. The GUI 1000 may be used toconfigure settings of the device and/or an audible assistant applicationundertaking present principles. Furthermore, it is to be understood thateach of the options or sub-options to be discussed below may be selectedby directing touch or cursor input to the adjacent check box shown foreach respective option.

As shown in FIG. 10, the GUI 1000 may include a first option 1002 thatis selectable to enable tailored audible assistance as described herein.For example, selection of the option 1002 may configure thedevice/application to undertake the functions and routines describedherein, such as the logic of FIGS. 8 and 9.

The GUI 1000 may also include an option 1004 with respective sub-options1006-1012 that are selectable for a user to provide input of the user'sobjective(s) while playing a video game so that the device/applicationmay tailor audible output/assistance to providing information that helpsthe user achieve his or her specific objective(s). As shown, thesub-options 1006-1012 are respectively for tailoring the output based onthe user achieving the most points possible or achieving a new highscore for the video game, achieving a best player rating for the videogame, getting the “full experience” of the video game, and completingvarious video game levels or chapters as fast as possible even if notgetting maximum points from each level/chapter).

Still further, the GUI 1000 may present an option 1014 that isselectable to configure the device/application to give specific andexplicit audible assistance describing how the user may achieve his orher objective(s), while an option 1016 may instead be selected toconfigure the device/application to give more-general hints to help theuser achieve his or her objective(s) without giving spoilers of upcomingfeatures of the video game or ruining the suspense of how to “unlock” oraccomplish something in the video game.

Providing additional disclosure in accordance with present principles,it is to be understood that APIs conforming to a video game consolemanufacturer's standards as indicated in a technology requirementchecklist (TRC) may be provided as part of a video game by the videogame's producer. However, non-conforming APIs may still be used such as,e.g., in embodiments where artificial intelligence software may he usedto process input from the non-conforming API to identify an AV contentlocation, relevant audible assistance to provide, etc.

As far as audible output/assistance goes, it is to be further understoodthat sometimes the output may include sarcastic or humorous comments toa user. An artificial intelligence system (AI) or deep learning modulemay be used to determine if sarcasm or humor is appropriate, and in factan AI system or deep learning module may be used as part of the audibleassistant application disclosed herein in order to make any of thedeterminations or identifications discussed herein as well as to tailorany audible output/assistance that is to be provided based on the AIsystem/module determining the output/assistance is relevant based onvarious factors as discussed herein. For example, an AI system employedto undertake present principles can analyze a single image frame of AVcontent to identify a location/segment within the AV content that iscurrently being presented, or the AI system may instead analyze, e.g., athree to five second span of image frames.

Regarding the audible assistant applications/devices disclosed herein,whether or not executed by an AI system, it is to be understood theassistant may be programmed to know how precise it should be inproviding audible assistance. For example, a video game provider orvideo game console manufacturer may customize the assistant or providecustomized data for use by the assistant based on a particular videogame or other type of AV content for which audible output is to beprovided. Additionally or alternatively, the assistant may be trained tocustomize its output dynamically based on, e.g., user feedback aboutwhether the output was good or bad.

Regarding using identification of a given object within image frames ofAV content n order to identify a location/segment of the AV contentbeing presented, object or asset identifications (IDs) may be used asassociated with various locations within the AV content. Furthermore,the assistant may be trained to recognize a given object from all anglesshould one video game player control the video game to view the objectfrom one angle while another video game player controls the same videogame to view the object from a different angle.

Furthermore, it is to be understood in accordance with presentprinciples that audible assistance/output may be provided for othertypes of AV content besides video games. For instance, if a user iswatching a movie or television show and wonders aloud what actor oractress is playing a particular character, a device undertaking presentprinciples may process the user's speech using natural languageprocessing to tailor an audible output that identifies the actor oractress.

Still regarding tailored audible outputs, in addition to what isdisclosed above, these outputs may be determined still other ways. Forexample, a provider of a given video game may provide a list of gameactions a user is to take or goals a user is to accomplish and at whatpoints in the game those actions/goals arise, and then a given audibleoutput may be tailored to indicate the action or goal to be accomplishedas indicated on that list once the associated location is reached. Thelist may even have various “hints” of different levels of specificity,with more general hints being provided and more specific hints thenbeing provided if, e.g., a user still struggles to accomplish a certaingoal even after receiving the more-general hint. The hints themselvesmay be pre-programmed by a video game console manufacturer or video gameproducer in still other ways and associated with various sections of thevideo game, and then such a hint may be selected for presentation to theuser once an associated section of the video game has been reached.What's more, hints may be crowdsourced based on difficulties video gameplayers at other locations have had or information/hints those playershave requested in the past even if, e.g., provided in natural languagerather than as voice command input. Still further, various players of avideo game may post questions or requests for hints to an online forumor help system, and then the console manufacturer or video game producermay select a best answer or provide its own and program that as a “hint”to be audibly provided by an audible assistant undertaking presentprinciples.

As another example, for major tasks of a video game (e.g., those forwhich a player cannot advance from one chapter to another withoutaccomplishing the task), a console manufacturer or video game providermay specify specific, tailored outputs that are to be provided when auser requests assistance or when such assistance is determined to berelevant based on a user's natural language. Then for other “smaller”tasks, machine learning may be used to identify and select anappropriate tailored response.

Still further, for identifying a segment/location within AV content thatis a video game specifically, in addition to using camera images, framebuffers, etc. as described above, identification of a video game playerexecuting a particular skill move of a predeterminedbutton-press/joystick combination using a video game controller may alsobe used to identify the location. This may be accomplished owing tovarious button-press/joystick combinations being built in to the videogame so that the player learns them at various different stages of thevideo game. Hence, if a combination that a player will not learn ordiscover until a later chapter of a video game is identified, a deviceundertaking present principles may determined that the current locationof the video game must be at least at that later chapter. Thesecombinations may also be used to identify a user's gameplay ability orthe difficulty level at which a video game has been set as describedabove so that more general hints can be provided for more advancedplayers and higher difficulty levels, while more specific hints can beprovided for less advanced players and lower difficulty levels, e.g., asprogrammed or predefined by a manufacturer or video game provider.

As may be appreciated from the foregoing detailed description, presentprinciples thus improve the functionality and ease of use the devicesdescribed herein and also improve the digital assistants those devicesmay execute through the technological solutions described herein.

It will be appreciated that whilst present principals have beendescribed with reference to some example embodiments, these are notintended to be limiting, and that various alternative arrangements maybe used to implement the subject matter claimed herein.

What is claimed is:
 1. A device, comprising: at least one processor; atleast one computer storage with instructions executable by the at leastone processor to: identify a location within audio video (AV) contentthat is currently being presented; and provide, based on theidentification of the location within the AV content, audible assistancepertaining to the AV content.
 2. The device of claim 1, wherein the AVcontent comprises a video game, and wherein the location comprises oneof: a chapter of the video game, a sub-chapter of the video game.
 3. Thedevice of claim 1, wherein the location is identified based at least inpart on images from a camera in communication with the at least oneprocessor, the images from the camera indicating a portion of the AVcontent.
 4. The device of claim 1, wherein the audible assistance isfirst audible assistance, wherein the location is a first location, andwherein the instructions are executable by the at least one processorto: provide the first audible assistance based on the identification ofthe first location meeting a threshold confidence level; and decline toprovide second audible assistance based on an identification of a secondlocation within the AV content not meeting the threshold confidencelevel.
 5. The device of claim 1, wherein the content of the audibleassistance varies based on a level of confidence in the identificationof the location.
 6. The device of claim 1, wherein the content of theaudible assistance is tailored to pertain the location based on a levelof confidence in the identification of the location being above athreshold confidence level, and wherein the content of the audibleassistance is not tailored to pertain to the location based on the levelof confidence in the identification of the location being below thethreshold confidence level.
 7. The device of claim 1, wherein thecontent of the audible assistance is determined based on execution ofnatural language processing on speech received from a user to identifyinformation relevant to what is spoken by the user, the speech beingreceived via a microphone in communication with the at least oneprocessor.
 8. The device of claim 1, wherein the content of the audibleassistance is tailored to a user's emotion(s) while observing the AVcontent, the emotion(s) being identified by the at least one processor.9. The device of claim 1, wherein the AV content comprises a video game,and wherein the content of the audible assistance is tailored to atleast one of: a player's gameplay ability, a difficulty level at whichthe video game is set.
 10. The device of claim 1, wherein the AV contentcomprises a video game, and wherein the content of the audibleassistance varies based on an objective of a player, the objectiveidentified by the at least one processor, the objective pertaining toone or more of: a game score, a player rating, a level of engagementwith various aspects of the video game, completing the video game. 11.The device of claim 1, wherein the device is a first device, and whereinthe first device comprises a robot that is different from a seconddevice at which the AV content is presented.
 12. The device of claim 1,wherein the device at least one of: is embodied in a video game console,communicates with the video game console to identify the location.
 13. Acomputer-implemented method, comprising: identifying a section ofcontent that is currently being presented via at least one device; andproviding, based on the section of the content that is currently beingpresented, audible output pertaining to the section.
 14. Thecomputer-implemented method of claim 13, wherein the section of contentis identified at least in part by executing natural language processingon speech received from a user via a microphone.
 15. Thecomputer-implemented method of claim 13, wherein the section of contentis identified based at least in part on at least one first image from acamera, the at least one first image itself indicating at least onesecond image of the section as presented on a display.
 16. Thecomputer-implemented method of claim 15, wherein the section of thecontent is identified from only partial visual images of the sectionthat are indicated in the at least one first image from the camera. 17.The computer-implemented method of claim 13, wherein the contentcomprises a video game, and wherein the audible output is tailored tohelp the user advance in the video game based on an objective of theuser that is identified.
 18. The computer-implemented method of claim 1,wherein information provided via the audible output is determined basedon a level of confidence in the identification of the location.
 19. Anapparatus, comprising: at least one computer memory that is not atransitory signal and that comprises instructions executable by at leastone processor to: identify a section of a video game that is currentlybeing presented via a device; and provide, based on the section of thevideo game that is currently being presented, audible output pertainingto the video game.
 20. The apparatus of claim 19, wherein the substanceof the audible output is determined based on a level of confidence inthe identification of the section and based on execution of naturallanguage processing on speech received from a player via a microphone incommunication with the at least one processor.